Research for reproducing a three-dimensional image has been carried out. According to one known method for reproducing a three-dimensional image, two pictures photographed of the same object in different directions are displayed such that the two pictures are laid side-by-side to make right and left eyes of a viewer look at the two pictures respectively. A pair of the two pictures used in this manner is called a stereo picture.
In order to generate a stereo picture, for example, one camera at a first position is used to photograph a picture of a target object to create a first picture, and then, the camera is moved to a second position to photograph a picture of the target object to create a second picture. Alternatively, two cameras respectively arranged at the first and second positions photograph pictures of the target object to generate the first and second pictures. The first and second pictures become a pair for a stereo picture.
The two pictures included in the stereo picture are viewed by right and left eyes of a viewer respectively. For this reason, in order to regenerate a three-dimensional image of high quality, the images captured on the two pictures are preferably photographed in the same condition as in a condition that a viewer generally looks at the object. Right and left eyes of a human are positioned along a horizontal direction. For this reason, the first and second positions are preferably arranged in the horizontal direction, as well. Respective sight lines of the right and left eyes are approximately in parallel. For this reason, an optical axis of the camera at the first position, and an optical axis of the camera at the second position are preferably in parallel.
To accurately arrange the cameras at the first and second positions in such a manner, strict position alignment is performed. For this purpose, positions and postures of the cameras at first and second positions are measured, and in accordance with the result of the measurement, arrangement of the cameras is adjusted. In order to perform such strict alignment adjustment, for example, a worker uses a measuring device to make measurement, and in accordance with the result of the measurement, the worker performs various manual works for position adjustment of the cameras. Thus, it is expensive and takes time to create a stereo picture. Accordingly, it is often difficult to perform the strict alignment adjustment. As a result, a height of the camera at the first position is different from a height of the camera at the second position, the optical axis direction of the camera at the first position is different from the optical axis direction of the camera at the second position, or one of the pictures is displaced from the other of the pictures in a rotational direction around the optical axis.
In view of the above-described matters, techniques of transforming an image position on a picture to create a picture suitable for a stereo picture has been proposed (for example, Japanese Laid-open Patent Publication No. 2008-186145, and Richard I. Hartley, “Theory and practice of projective rectification” International Journal of Computer Vision, P. 115-127, No. 2, Vol. 35, 1999, November). According to such one of the techniques, by taking into consideration a property of the camera such as a focal length and distortion aberration of the camera, image positions on the picture is transformed so as to correct differences between heights, optical axis directions, and positions in the rotational direction around the optical axes of the cameras at the two positions at the time of photographing.
A picture processing device according to one example of such known techniques determines a rotational transformation equation or a rotational and translational transformation equation of plane projective transformation so that the corresponding feature points respectively extracted from the two pictures match each other. Then, the picture processing device uses the rotational transformation equation or the rotational and translational transformation equation to transform each position on the picture to generate a stereo picture.
A picture processing device according to another known technique transforms a picture to generate a stereo picture by taking epipolar geometry into consideration. When one target point of an object is photographed from two different picture-photographing positions, the epipolar geometry represents corresponding relationship on the picture between the target point and each of the photographing positions. Specifically, a line connecting the target point and the camera arranged at the first position is presented by a straight line when projected on the picture photographed by the camera arranged at the second position. Similarly, a line connecting the target point and the camera arranged at the second position is presented by a straight line when projected on the picture photographed by the camera arranged at the first position. The straight line projected on the picture is referred to as epipolar line. The device using the epipolar geometry takes such corresponding relationship into consideration, and obtains plane projective transformation coefficients for making the epipolar lines in parallel to transform at least one of the pictures. Thereby, it is possible to artificially create a pair of two pictures that are photographed in a state that the optical axes of the cameras are in parallel.
According to the technique of using the epipolar geometry, when the feature point on each picture corresponding to the target point is not accurately obtained, due to an error in extracting the feature point, accurate plane projective transformation coefficients may not be calculated. Further, according to this technique, due to picture distortion such as distortion aberration of an imaging optical system of the camera, accurate plane projective transformation coefficients may not be calculated.
According to a technique of obtaining the coefficients for the rotational transformation equation or the rotational and translational transformation equation such that the corresponding feature points between the two pictures match each other, a positional error is minimized for an entire of the plural pairs of the corresponding feature points. Accordingly, in this technique, even when the pairs of the corresponding feature points includes several erroneously correlated pairs of the feature points, relatively accurate coefficients can be obtained.
However, when an object has a certain depth in a front-to-back direction, a distance between the feature points that are extracted from the two pictures and that correspond to the points on the front side of the object is larger than a distance between the feature points that are extracted from the two pictures and that correspond to the points on the back side of the object. For this reason, in order to perform accurate transformation of image positions on the picture such that the pairs of the corresponding feature points on the two pictures match each other, nonlinear position transformation is normally used. The nonlinear position transformation transforms each position on the picture so as to adjust a transfer amount in accordance with a length to the object corresponding to the position.
Meanwhile, the positional transformation performed by the rotational transformation equation or the rotational and translational transformation equation is a linear transformation transforming a straight line into another straight line. Accordingly, when the object has a length in the front-to-back direction, in a picture transformed by the plane projective transformation coefficients determined on the basis of the feature points on the object, the image of the object may be distorted due to the transformation.